1. Field of Invention
This invention relates to neural networks, and to communication systems, which make use of them.
2. Description of the Related Art
Neural networks were developed, over the past half century, as a method of computing by attempting to emulate the arrangement of biological neurons. Neural networks therefore, in general, perform processing by combined a number of parallel, simple, calculations. The main use of neural networks is as a learning architecture. In this use, the network is “trained” by applying data to the input of the network.
Neural networks may be implemented as parallel processing hardware (using electronic, opto-electronic, optical, or other computing elements), but are more normally implemented by using one or more conventional computers to perform the calculations of each of the neurons. Thus, whereas the literature discusses “neurons” and calculations in “parallel”, in fact a computer implements these as sequential calculations.
The best known, and most widely used, neural network is the “multi-layer perceptron” (MLP). MLPs are usually trained using the “back-propagation algorithm” developed by Rumelhart et al (D. E. Rumelhart, G. E. Hinton and R. J. Williams, Learning internal representation by error propagation, in: D. E. Rumelhart and J. L. McClelland, Eds., Parallel Distributed Processing:Explorations in the microstructure of cognition, Vol. 1: Foundations, Chapter 8 (MIT Press, 1986)). This method may also be used in embodiments of the present invention.
In a multi-layer perceptron, each node is a feed-forward node, with a number of inputs, a number of weights, a summation point, a non-linear function, and an output port. Each input at an input port is multiplied by the value of the corresponding weight, and the weighted values are summed. In other words, the input vector is multiplied by the weight vector to form a scalar product.
The sum is then fed through a non-linear function (typically a sigmoid function) and passed to the output port. In a multi-layer perceptron, at least two layers are present, so that the output ports of the first layer are connected to the input ports of a second layer.
Neural networks can be trained to recognise patterns, which is their most common application. However, they can also be trained to learn (and therefore replicate) arithmetic or algorithmic functions. In either case, the network is “trained” by applying a pattern of inputs to the inputs of the network, calculating the difference between the outputs at the output nodes of the net and the desired outputs, and using the difference to modify the weight values of the net.
One function for which neural networks have been proposed is decoding of convolutional codes used in signal transmission as error correcting transmission codes, or error detecting codes. Such codes are used to encode a time-varying signal comprising a sequence of input symbols, which is coded such that each coded output symbol depends on more than one input symbol, typically by providing more than one parallel processing branch each having symbol delay stages, and generating, on each branch, modulo-2 sums of several input symbols. Various types of decoder are known, of which the widely-used Viterbi decoder is the best known.